Fix Agent close hanging by JoTurk · Pull Request #934 · pion/ice

JoTurk · 2026-06-30T22:46:47Z

Description

Abort started candidate i/o before waiting for the task loop to stop.
this prevents close from deadblocking when a task is blocked in WriteTo
This is a fix i have been testing for a few days it's missing regression test and some improvements.

JoTurk · 2026-06-30T22:47:14Z

@boks1971 I'll get this improved ready for review + regressions tests by tomorrow.

codecov · 2026-06-30T22:50:17Z

Codecov Report

❌ Patch coverage is 70.08547% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.10%. Comparing base (3ecc088) to head (24d50e5).

Files with missing lines	Patch %	Lines
udp_mux.go	54.68%	20 Missing and 9 partials ⚠️
candidate_base.go	68.42%	3 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #934      +/-   ##
==========================================
- Coverage   88.33%   88.10%   -0.23%     
==========================================
  Files          46       46              
  Lines        5973     6071      +98     
==========================================
+ Hits         5276     5349      +73     
- Misses        482      496      +14     
- Partials      215      226      +11

Flag	Coverage Δ
go	`88.10% <70.08%> (-0.23%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

JoTurk · 2026-07-06T04:17:39Z

@boks1971 the new tests in agent_test deadblock on main. which confirms the fix i think also with the results from your dev env.

Copilot

Pull request overview

This PR addresses an Agent shutdown deadlock by aborting started candidate socket I/O before waiting for the task loop’s active task to return, preventing Close() from hanging when a task is blocked in WriteTo.

Changes:

Add CloseWithPreStop to taskloop.Loop and use it from Agent.close() to run a pre-stop hook before waiting for the loop to stop.
Track started candidates in Agent and abort their I/O during shutdown to unblock in-flight socket operations.
Add abort-write plumbing for muxed/shared packet conns and introduce regression tests for blocked candidate writes.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
udp_muxed_conn.go	Adds `abortWrite()` forwarding to the UDP mux to help unblock muxed writes during shutdown.
udp_mux.go	Implements `abortWrite()` via write deadlines to interrupt blocked `WriteTo` on the shared UDP socket.
shared_packet_conn.go	Adds `abortWrite()` passthrough when the underlying conn supports write abortion.
internal/taskloop/taskloop.go	Introduces `CloseWithPreStop` and `sync.Once` to ensure close/preStop are executed once safely.
internal/taskloop/taskloop_test.go	Adds a concurrency test for `CloseWithPreStop` behavior (currently contains a compile issue).
candidate_base.go	Refactors candidate close into `abortIO()` + wait, and adds write-abort support via `writeAborter`.
agent.go	Tracks started candidates and aborts their I/O as a pre-stop hook during agent shutdown.
agent_test.go	Adds regression tests ensuring Agent close unblocks blocked writes (direct, shared, and UDP mux).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

JoTurk · 2026-07-06T04:22:38Z

+	var wg sync.WaitGroup
+	wg.Add(closers)
+	for range closers {
+		go func() {
+			defer wg.Done()
+			loop.CloseWithPreStop(func() {
+				preStopCalls.Add(1)
+			})
+			closeReturned <- struct{}{}
+		}()
+	}


cnderrauber

Looks good, thank you!!

boks1971

lgtm!

But, the CoPilot flagged UDPMux seemed like something legit?

boks1971 · 2026-07-06T04:32:41Z

+	close(releaseBlock)
+	wg.Wait()
+
+	assert.Equal(t, int32(1), preStopCalls.Load())


a neat test!!!

JoTurk · 2026-07-06T04:46:05Z

@boks1971

But, the CoPilot flagged UDPMux seemed like something legit?

Yeah I'm adding an atomic counter.

Abort started candidate i/o before waiting for the task loop to stop. this prevents close from deadblocking when a task is blocked in WriteTo

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

JoTurk · 2026-07-06T05:10:24Z

@boks1971 i added the atomic counter, can you please review / test this again, i'll manually test this a lot before i merge it.

boks1971 · 2026-07-06T06:00:32Z

@boks1971 i added the atomic counter, can you please review / test this again, i'll manually test this a lot before i merge it.

Thank you @JoTurk . The changes lgtm! The Gosched calls are always a bit scary. Guess the alternate is not very useful here. Would be good to get 👀 from @cnderrauber again as he has spent a bunch of time in these mux modules.

My set up seems to be in a flux state currently. I will have to try and repro the failure I had before. But, please do not wait for me. I will report back issues when I get a chance to try.

cnderrauber · 2026-07-06T06:41:28Z

Is there a problem if we just left udpmux unchanged (or empty implementation of abortWrite)? The udp write will not be blocked, so we don't need to abort the write ?

boks1971 · 2026-07-06T06:47:05Z

Is there a problem if we just left udpmux unchanged (or empty implementation of abortWrite)? The udp write will not be blocked, so we don't need to abort the write ?

The original theory that forced these set of changes from Jo was that the STUN ping got blocked on a close of an ICE agent. Looks like golang can block UDP write, but has to be very specific conditions like kernel buffer exhaustion which I think should be very rare. But, that was the best theory for hangs in LiveKit server on participant close (especially participants who were not disconnecting cleanly).

cnderrauber · 2026-07-06T12:42:49Z

Sounds good. I have been concerned the loop hang on close when the write is blocked before, but thought the udp socket would not be blocked. The change looks good to me!

JoTurk · 2026-07-06T15:31:12Z

The original theory that forced these set of changes from Jo was that the STUN ping got blocked on a close of an ICE agent. Looks like golang can block UDP write, but has to be very specific conditions like kernel buffer exhaustion which I think should be very rare.

Yes I tried to reproduce it with stressing writes, and I think it has to do with OS buffer, and go handling of EAGAIN, I think it's a bug at the Go runtime, there are a few reports that I found golang/go#73919 golang/go#61555 I'm not sure why it happens more on darwin but i was able to replicate it once on linux.
I'll make that this doesn't cause a regression before i merge it and I'll look into fixing it at the go runtime.

boks1971 mentioned this pull request Jul 2, 2026

Catastrophic heisenbug goroutine leak in LiveKit 1.12+ livekit/livekit#4591

Open

JoTurk force-pushed the jo/fix-agent-close-hang branch 2 times, most recently from 82125a2 to 5168f71 Compare July 6, 2026 04:13

JoTurk marked this pull request as ready for review July 6, 2026 04:14

JoTurk requested review from boks1971 and cnderrauber July 6, 2026 04:14

JoTurk force-pushed the jo/fix-agent-close-hang branch from 5168f71 to 57a11b7 Compare July 6, 2026 04:15

JoTurk requested a review from Copilot July 6, 2026 04:17

Copilot started reviewing on behalf of JoTurk July 6, 2026 04:18 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

cnderrauber approved these changes Jul 6, 2026

View reviewed changes

boks1971 approved these changes Jul 6, 2026

View reviewed changes

Fix Agent close hanging

24d50e5

Abort started candidate i/o before waiting for the task loop to stop. this prevents close from deadblocking when a task is blocked in WriteTo

JoTurk force-pushed the jo/fix-agent-close-hang branch from 57a11b7 to 24d50e5 Compare July 6, 2026 04:55

JoTurk requested a review from Copilot July 6, 2026 04:56

Copilot started reviewing on behalf of JoTurk July 6, 2026 04:56 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Comment thread udp_mux.go

Comment thread udp_mux.go

Uh oh!

Conversation

JoTurk commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

JoTurk commented Jun 30, 2026

Uh oh!

codecov Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JoTurk commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

JoTurk Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cnderrauber left a comment

Choose a reason for hiding this comment

Uh oh!

boks1971 left a comment

Choose a reason for hiding this comment

Uh oh!

boks1971 Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

JoTurk commented Jul 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

JoTurk commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boks1971 commented Jul 6, 2026

Uh oh!

cnderrauber commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boks1971 commented Jul 6, 2026

Uh oh!

cnderrauber commented Jul 6, 2026

Uh oh!

JoTurk commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

JoTurk commented Jun 30, 2026 •

edited

Loading

codecov Bot commented Jun 30, 2026 •

edited

Loading

JoTurk commented Jul 6, 2026 •

edited

Loading

JoTurk Jul 6, 2026 •

edited

Loading

JoTurk commented Jul 6, 2026 •

edited

Loading

cnderrauber commented Jul 6, 2026 •

edited

Loading

JoTurk commented Jul 6, 2026 •

edited

Loading